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METHOD AND SYSTEM FOR ENSURING ROBUSTNESS IN NATURAL 
LANGUAGE UNDERSTANDING 

BACKGROUND OF THE INVENTION 

5 

1 . Field of the Invention 

The present invention relates to computer systems with 
natural language understanding capabilities, and more 
particularly to a method and system for ensuring robustness 
10 for these systems. 

2 . Description of the Related Art 

For computer systems with natural language 
understanding (NLU) capabilities, errors can be made by the 

15 system in translating the user's input, resulting in an 

incorrect action being executed by the system. Presently, a 
typical natural language understanding system which receives 
a command which is incorrect carries out the command. If 
the system carries out this action, problems may be 

20 encountered. For example, data may be changed, memory 

updated or erased, or other detrimental events may occur. 
The occurrence of these events may require undoing the 
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previous command or redoing a sequence of commands to return 
the system to a desired state. This results in lost time 
and annoyance of the user. 

Therefore, a need exists for a system and method for 
5 ensuring robustness in natural language understanding by 

determining incorrect commands and preventing their 
execution. 



SUMMARY OF THE INVENTION 

10 A method, which may be implemented by employing a 

program storage device readable by machine, and tangibly 
embodying a program of instructions executable by the 
machine to perform method steps for ensuring robustness of a 
natural language understanding (NLU) system, includes 

15 tagging recognized words of a command input to the NLU 

system to associate the command with a context, and 
translating the command to at least one formal command based 
on the tagged words. A top ranked formal command is 
determined based on scoring of the tagged recognized words 

20 and scoring translations of the at least one formal command. 

Whether the top ranked formal command is accepted is 
determined by comparing a feature vector of the top ranked 
Y0999-331 (8728-310) -2- 



formal command to representations of feature vectors stored 
in an accept model. The top ranked formal command is 
executed if accepted and incorrect commands are prevented 
from execution to provide a robust NLU system. 

In other methods, which may be implemented by 
employing the program storage device, the step of 
determining a top ranked formal command may include the step 
of ranking formal commands based on a product of scores of 
the tagged words and scores of translations of the at least 
one formal command. The step of determining a top ranked 
formal command may include the step of ranking N formal 
commands where N is a selectable system parameter. The step 
of determining whether the top ranked formal command is 
rejected by comparing the feature vector of the top ranked 
formal command to feature vector representations stored in a 
reject model may be included. The step of providing the 
reject model by including representations of feature vectors 
of formal commands corresponding to words or sentences to be 
rejected may also be included. 

The reject model may include a cluster of models. The 
step of clustering the cluster of models based on at least 
one of mistakes in commands, mistakes in arguments of the 
Y0999-331 (8728-310) -3- 



command, and processing mistakes may further be included. 
The method steps may include providing the accept model by 
including representations of feature vectors of formal 
commands corresponding to words or sentences to be accepted. 
The step of determining whether the top ranked formal 
command is accepted may include the step of computing a 
probability of acceptance for the command. The step of 
computing a probability of acceptance for the command may 
include the steps of computing a probability of rejection 
for the command and comparing the probability of acceptance 
to the probability of rejection to determine if the command 
is to be executed. The step of computing a probability of 
acceptance for the command may include the step of comparing 
the probability of acceptance to a threshold probability to 
determine if the command is to be executed. The threshold 
may be modified by the user. 

In other method steps, the accept model may include a 
cluster of models. The step of clustering the cluster of 
models based on at least one of mistakes in commands, 
mistakes in arguments of the command, and processing 
mistakes may be included. The step of preventing incorrect 
commands from execution to provide a robust NLU system, may 
Y0999-331 (8728-310) -4- 



include executing a do nothing command responsive to the 
incorrect commands . 

A method for building an evaluation corpus for checking 
commands in a natural language understanding (NLU) system 
includes providing a training corpus of words and sentences. 
The words and sentence have a user input form and a 
corresponding formal command associated with the user input 
form. At least some of corresponding formal commands 
include a do nothing command for incomplete and/or incorrect 
commands. The words and sentences of the training corpus 
are passed to the natural language understanding system to 
determine a top ranked command. The top ranked command is 
compared to the corresponding formal command to determine if 
a match exists. If a match exists, the word or sentence is 
placed in the accept corpus, otherwise in the reject corpus. 
Features from the words or sentences of the accept corpus 
and the reject corpus are extracted to construct a feature 
vector for each word or sentence, and an accept model and a 
reject model are constructed from the extracted feature 
vectors . 

In other methods, the feature vectors may include 
tagging scores for recognized words of the word or sentence 
Y0999-331 (8728-310) -5- 



represented by the feature vectors or translation scores for 
formal commands associated with the word or sentence 
represented by the feature vector. The feature vectors may 
include a do nothing score associated with words and 

5 sentences. The do nothing score indicates a probability 

that the do nothing command is present for associated words 
and sentences. The feature vectors may include a top 
command similarity measure for counting identical formal 
commands, and/or a parameter mismatch feature for measuring 

10 a number of command arguments in a translation of a command. 

The method may further include the step of clustering 
feature vectors according to selected characteristics and 
conditions to provide at least one of a cluster of accept 
models and a cluster of reject models. The accept model and 

15 the reject model may consist of mean vectors and covariance 

matrices of feature vectors representing the words and 
sentences and a number of words and sentences stored in the 
model . 

A natural language understanding (NLU) system includes 
20 a tagger adapted for tagging recognized words of a command 

input to the NLU system to associate the command with a 
context, and a translator adapted for translating the 
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command to at least one formal command based on the tagged 
words, A robustness checker is included for determining a 
top ranked formal command based on scoring of the tagged 
recognized words and scoring translations of the at least 
5 one formal command. The robustness checker determines 

whether the top ranked formal command is accepted by 
comparing a feature vector of the top ranked formal command 
to feature vector representations stored in an accept model . 
A command executor executes the top ranked formal command if 

10 accepted and prevents incorrect commands from execution to 

provide a robust NLU system. 

In alternate embodiments, the top ranked formal command 
may be ranked based on a product of scores of the tagged 
words and translation scores of the at least one formal 

15 command. The top ranked formal command may be selected from 

a ranked list of N formal commands where N is a selectable 
system parameter. An accept model for storing feature 
vector representations of accept commands may be included, 
and the feature vector representations may be employed to 

20 determine an acceptance probability of formal commands. The 

accept model may include a cluster of models. 
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The system may include a reject model for storing feature 
vector representations of reject commands. The feature 
vector representations are employed to be compared to a 
rejection probability of formal commands. The reject model 
may include a cluster of models. The robustness checker may 
include a feature extractor for extracting feature vectors 
from the command input , The feature vectors may include 
tagging scores for recognized words of a word or sentence 
represented by the feature vector and/or translation scores 
for formal commands associated with the word or sentence 
represented by the feature vector. The feature vectors may 
also include a do nothing score associated with words and 
sentences. The do nothing score for indicates a probability 
that a do nothing command is present for associated words 
and sentences. The do nothing command is associated with 
incomplete or incorrect commands which may be input as a 
command. The feature vectors may include a top command 
similarity measure for counting identical formal commands 
and/or a parameter mismatch feature for measuring a number 
of command arguments in a translation of a command. The 
robustness checker may include a robustness evaluator for 
determining whether the top ranked formal command is 
Y0999-331 (8728-310) -8- 



accepted by comparing the top ranked formal command to 
feature vectors stored in an accept model - 
3 A natural language understanding system includes a 

corpus of rejectable commands corresponding to incorrect 
commands capable of being input by a user and do nothing 
commands corresponding to the incorrect commands input to 
the system which maintain the system in an idle state and/or 
8 prompt the user to input a response. 

3 These and other objects, features and advantages of the 

■-3 present invention will become apparent from the following 

detailed description of illustrative embodiments thereof, 
which is to be read in connection with the accompanying 
:=:13 drawings. 

Q BRIEF DESCRIPTION OF DRAWINGS 

The invention will be described in detail in the 
following description of preferred embodiments with 
40 reference to the following figures wherein: 

FIG. 1 is a block/flow diagram of a system/method for a 
natural language understanding (NLU) system employing a 
robustness checker in accordance with the present invention; 
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FIG- 2 is an illustrative example of output from a 
2 tagger and a translator in accordance with the present 

invention; 

FIG. 3 is a block/flow diagram illustratively showing 
the construction of an accept corpus and a reject corpus in 
accordance with the present invention; 
7 FIGS. 4A and 4B are block/flow diagrams showing a 

system/method for constructing accept and reject models in 
accordance with the present invention; 

FIG. 5 is a block/ flow diagram showing a robustness 
checker in accordance with the present invention; 
12 FIGS. 6A and 6B are block/flow diagrams showing a 

system/method for clustering accept and reject models in 
accordance with the present invention; and 

FIG. 7 is a block/flow diagram showing a robustness 
checker employing clustered models in accordance with the 
17 present invention. 



DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

The present invention ensures robustness in computer 
systems by automatically recognizing the inputs that are 
likely to result in an incorrect action, and taking 
YO 999-331 (8728-310) -10- 



preventive measures. For natural language systems, where a 
command is misinterpreted or incorrectly given, the present 
invention determines the incorrectness of the command prior 
to its execution and takes appropriate action. For example, 
incorrectly interpreted utterances are identified and mapped 
to a "do nothing" command, and thus the execution of a 
potentially incorrect command is avoided. The system can 
remain idle for a do nothing command or the system may 
prompt a user for more information. 

Although confidence scoring for speech recognition 
systems is employed in recognizing speech, these solutions 
do not address the errors caused by the natural language 
understanding systems, do not use feature extraction based 
on natural language understanding, and do not have a "do 
nothing" command built-in explicitly into the system. These 
solutions are also specific to systems with spoken input 
only, and are not relevant to systems where the input may 
have come from another source, such as a typed input or a 
handwritten input. The present invention accommodates input 
from a plurality of different sources for example, 
handwriting recognition devices, speech recognition systems, 
typed text. Input devices may include a variety of devices 
Y0999-331 (872 8-310) -11- 



such as, telephones, personal digital assistants, computers, 
etc . 

The invention provides a method and apparatus to build 
a statistically trainable system, which will be called a 
robustness checker, hereinafter, that is part of a natural 
language understanding system, and is capable of recognizing 
user inputs that are likely to have been incorrectly 
translated by a natural language understanding (NLU) system. 
The robustness checker is statistically trained using the 
same training data that is used to build models for the NLU 
system. The NLU system may include a tagger component and a 
translator component. When deployed as part of a real-time 
conversational interface, the NLU systems will produce the 
best choice, along with N-1 other choices (for a total of N 
choices) for the formal command corresponding to the user's 
input. The training data will include utterances for which 
no legal formal language statement exists, and these 
utterances will be mapped to a "do_nothing" command. 

All the training data that was used to build the NLU 
models are subjected to the NLU, and on the basis of the 
results, the training sentences may be divided into two 
categories: the accept category, and the reject category. 
Y0999-331 (8728-310) -12- 



The accept category includes all the sentences for which the 
NLU's first choice of formal command is correct, and the 
reject category includes all the sentences for which the 
first choice of formal command is not correct. 
5 A feature vector is computed for each of the 

utterances. The feature vector may include one or more of 
the following: tagger scores, translator scores, the 
normalized score for the "do_nothing" command for each 
tagging, a command similarity measure of the N choices of 

10 formal commands from the NLU, and a parameter mismatch 

factor between the tagger output and the translator output. 
For each of the accept and reject categories, statistical 
models are built using the feature vectors from that 
category. These models are used by the robustness checker 

15 to make a decision on whether to accept or reject a formal 

command produced by the NLU corresponding to a new user 
input. When appropriate, more than one model may be built 
for each category to allow for clustered modeling. 

When deployed as part of a real-time system, the 

20 robustness checker will first calculate the feature vector 

corresponding to each input sentence from the user, and map 
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the formal command produced by the NLU for that input 
sentence to either the accept category or the reject 
category. If the formal command is mapped to the accept 
category, then the formal command is submitted for 
5 execution. If the command is mapped to the reject category, 

then it is rejected and the user may be asked to repeat the 
input . 

It should be understood that the elements shown in 
FIGS, 1 and 3-7 may be implemented in various forms of 

10 hardware, software or combinations thereof. Preferably, 

these elements are implemented in software on one or more 
appropriately programmed general purpose digital computers 
having a processor and memory and input/output interfaces. 
Referring now to the drawings in which like numerals 

15 represent the same or similar elements and initially to FIG. 

1, a block/flow diagram an NLU system 10 and method 
employing a robustness checker is shown in accordance with 
the present invention. A user (or users) submits an input 
to the system 10 via user interface 10 0. The input from 

20 user may be in text, either typed or recognized from speech 

or recognized or generated from another medium. The input 
may also be in spoken/acoustic form or handwritten form. If 
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the input is in text form, such as an input submitted using 
a computer keyboard or personal digital assistant/telephone 
keypad, then the text is sent to tagger 200. If the input 
is not in text form, such as a spoken input or a handwritten 
5 input, then the input (from a speaker, acoustic signal or 

visual signal) is translated to text form using speech 
recognition or handwriting recognition in block 150. The 
recognized text is then sent to tagger 200, Techniques for 
speech recognition and handwriting recognition may be 

10 employed as known in the art . 

Tagger 2 00 is responsible for recognizing classes of 
words and phrases present in the input sentence. For 
example, in a natural language interface to an electronic 
mail application, the user^s input may be: 

15 forward this message to David 

The corresponding output from tagger 2 00 may be: 

forward this message to NAME 
where "David" has been tagged as "NAME". Tagger 200 may be 
built using statistical parsing techniques, such as those 

20 described in Ward, T., et al . , "Towards Speech Understanding 

Across Multiple Languages," International Conference on 
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Spoken Language Processing, Sydney, Australia, December 
1998, incorporated herein by reference. 

The output from tagger 200 is sent to translator 300 
which will assign the tagged sentence to a formal command. 
5 For example, the illustrative sentence described above may 

be assigned to a formal command of the form: 

f orward_message (message=current , recipient=NAME) 
Translator 2 00 may be built using the techniques described 
}4 in Papineni, K. , et al . , "Feature-Based Language 

%10 Understanding", Eurospeech, Rhodes, Greece, September 1997, 

SI incorporated herein by reference. 

Referring to FIG. 2, an illustrative example of the 
y3 output from tagger 200 and translator 3 00 is shown. In this 

IJ example, the user's input is "close the message log folder", 

y3l5 and tagger 200 produces three possible tagged outputs 202, 

and associated scores 2 04. For example, the first choice of 
tagger 200 is "CLOSE the FNAME FOLDER" with a score of 
0-873911, which also happens to be the correct tagging for 
this sentence. In this example, for each of the three 
20 tagged sentences, the translator 3 00 produces five choices 

of formal commands 302 and associated scores 304, The first 
choice of translator 3 00 for the first tagged sentence is 
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" close__f older { name=FNAME )" with a score of 0,96887374, 
which also happens to be the correct translation for this 
sentence. The selected formal command is " close_f older ( 
name="message log" ), with a final score of 0,846709646214, 
which is calculated as the multiplicative product of the 
scores from tagger 200 and translator 300. In this example, 
tagger 2 00 produces three choices, and translator 3 00 
produces five choices for each of the three choices from 
tagger 200, for a total of fifteen choices. In the general 
case, there will be N choices of formal commands, and the 
value of N is a system parameter. 

In one embodiment of the invention, N is fifteen, as 
in the example given in FIG. 2. The final choice of the 
formal command is selected using the tagger output and the 
translator output for which the combined multiplicative 
score is the highest, and this command will be called the 
top ranking formal command. 

Returning to FIG. 1, the output from the tagger 200 and 
translator 300, which may be of the form given in FIG. 2, is 
sent to a robustness checker 400. The robustness checker 
400 is responsible for determining if the top ranking formal 
command should be accepted or rejected. If the command is 
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to be accepted, then it is sent to command executor 50 0 and 
the command is executed. If the command is to be rejected, 
then the command is not executed and the user may have to 
resubmit the input, perhaps using a different choice of 
words . 

Referring to FIG. 3, an illustrative block/flow diagram 
shows building accept and reject corpora, according to the 
present invention. A training corpus 601 includes data that 
is used for training tagger 200a and translator 300a models. 

The contents of training corpus 601 include both input 
sentences and the associated formal commands. In a natural 
language interface for an electronic mail application, for 
example, the contents of training corpus 601 may 
illustratively include the following: 

do I have any new mail // check_new_mail ( ) 

forward this to David // f orward_message {message=current , 
recipient =David) 

do I have any uh err // do_nothing{) 

it is a nice day // do_nothing() 

The left side (before the "//" sign) includes the actual 
input sentences from the user, and the right side includes a 
Y0999-331 (872 8-310) -18- 



corresponding correct formal command. In accordance with 
the present invention, some of the sentences are mapped to a 
"do_nothing" command, either because they are clearly out of 
the domain, or because they are not complete enough to form 
a command, etc. The "do_nothing" command is one way 
robustness is ensured for the natural language system 10. 
All of the sentences in the training corpus 601 are 
subjected to tagger 200a and translator 300a. Tagger 200a 
is functionally equivalent to tagger 200 from FIG. 1, and 
translator 300a is functionally equivalent to translator 300 
from FIG. 1. For each sentence in the training corpus 601, 
the output of the translator 3 00a is examined by the 
translation checker 600. If the output is correct (i.e. the 
correct formal command is selected by the system which 
corresponds to the correct command in the training corpus) , 
then the sentence is added to the accept corpus 602. If the 
output is incorrect, then the sentence is added to the 
reject corpus 6 03. 

Referring to FIGS. 4A and 4B, an example of the process 
and system for building accept and reject models are 
illustratively depicted, according to the present invention. 
Each of the sentences in the accept corpus 602 and the 
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reject corpus 603 are subjected to feature extractors 401a 
and 401b, respectively, both of which are functionally 
equivalent to each other. The feature extractors 4 01a and 
401b are responsible for extracting a set of features for 
5 each sentence, and constructing a feature vector, v. 

In one embodiment of the invention, the following 
features are used to construct the feature vector, v. All 
tagging scores and all translation scores from the output 
are included (see FIG. 2) in the feature vector, v. The 
10 next j features are the normalized cumulative do_nothing 

scores for taggings 1 through j , where j is the total number 
of taggings. The do_nothing score for tagging j could be 
written as 

'■=i'^-^^ .... (1) 

15 where lij is the translation rank of the ith do_nothing 

command in tagging and kj is the number of do_nothing 
translations of tagging j. In the example of FIG. 2, three 
taggings are given. DNSi, DNS2 and DNS3 are calculated. The 
do_nothing score increases when more do_nothing translations 

20 are present in a tagging or when a do__nothing translation is 
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assigned a higher ranking in the translations of a tagging. 

Another feature is a top command similarity measure, TCS. 
This TCS feature counts the number of formal commands that 
are identical to the top command of the top tagging, without 
considering the arguments of the command. For the example 
in FIG. 2, close__f older ( name=FNAME ) and 
close_folder ( folder =current ) would be considered 
identical. This feature can be written as 



where ki is the index of the ith translation that is 
identical to the first, and k is the number of translations 
identical to the first tagging. 

Another feature includes a parameter mismatch feature, 
which measures the number of command arguments that are 
present in the translation of a command, but are not 
available in the tagging of that command. For example, in 
FIG. 2, where the selected formal command is close_f older 
(name=FNAME ) and the argument FNAME maps to "message log", 
the parameter mismatch would be 0 . On the other hand, if 




k 



(2) 
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none of the words were tagged as FNAME then argument will be 
incomplete, and in this case the parameter mismatch would be 
1. For commands with more than one argument, the parameter 
mismatch may be greater than 1. 
5 The feature extractor 401a extracts the features and 

constructs the feature vector for each sentence and submits 
the feature vector to a model constructor 604a. Similarly, 
the feature extractor 401b submits the feature vector to 
y model constructor 604b. Model constructors 604a and 604b 

plO are functionally equivalent to each other. In one 

C:3 embodiment of the invention, model constructor 604a computes 

an accept mean vector mA and accept covariance matrix Sa for 
^ all the feature vectors produced by the feature extractor 

401a, corresponding to all the sentences in the accept 
^""15 corpus 602. An accept model 605 includes the accept mean 

vector mA, the accept covariance matrix Sa, and the 
total number of sentences nA in the accept corpus 602. 
Similarly, model constructor 604b computes a reject mean 
vector mR and reject covariance matrix Er for all the 
20 feature vectors produced by the feature extractor 4 01b, 

corresponding to all the sentences in the reject corpus 603. 
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The reject model 605 includes the reject mean vector mR, the 
reject covariance matrix Er, and the total number 
of sentences nR in the reject corpus 603. 

Referring to FIG. 5, a robustness checker 400 is 
5 schematically shown in accordance with the present 

invention. Robustness checker 400 connects to translator 
300. Robustness checker 400 include feature extractor 401, 
robustness evaluator 4 02, accept model 605 and reject model 
606. Feature extractor 401 is functionally equivalent to 

10 feature extractors 401a and 401b described previously with 

reference to FIGS. 4A and 4B. 

Robustness evaluator 402 is responsible for evaluating 
a given feature vector, calculated by feature extractor 401 
using the output from translator 3 00 for a new sentence from 

15 the user, and determining if the corresponding formal 

command should be accepted (i.e. submitted for execution), 
or rejected. 

The robustness evaluator 402 first calculates P (A) , the 
prior probability for command acceptance, and P(R), the 
20 prior probability for command rejection. In one embodiment 

of the invention, P (A) and P (R) are calculated using 

P(A) = Ua / (Ha + ns) ... (3) 
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P(R) = Hr / (Ha + Hr) 



(4) 



10 



where Ha and Hr are the number of sentences in the accept 
corpus 602 and reject corpus 603, respectively. 

Given a feature vector v produced by the feature 
extractor 401, the robustness evaluator 4 02 calculates the 
conditional probabilities P{v|A) and P(v|R), using 



P(v A) = 



-|(v-'«a)^5:a(v-/wa) 



(2;r)2 eJ^ 



(5) 



4(V-/«r)'^Zr'(V-/«r) 



P(v R) = 



(6) 



T represents the transpose operator for vector v-iiia and 
S"^ is an inverse matrix. The classification rule is 
simple. A formal command with feature vector v is accepted 
if: 



15 



P{A)P(v|A) > P(R)P{v|R) 



... (7) 



and rejected otherwise. If the command is to be executed, 
then it is submitted to a command executor 500. 
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Variations to the classification mechanism specified by 
Equations (3) -(7) above will now be described in greater 
detail. One variation is to permit the user to modify the 
values of P (A) and P(R). In one embodiment of the 
invention, the user interface 100 (FIG. 1) permits the user 
to modify the values of P (A) and P (R) . By changing the 
values of P (A) and/or P (R) , users can modify the system 
behavior to suit desired preference. For example, 
it the user feels that too many commands are being rejected 
by the system, then increasing the value of P (A) (or 
equivalently, decreasing the value of P(R)) would increase 
the acceptance rate of the commands. An extreme case is 
when the value of P(R) is set to 0, where all commands will 
be accepted. Once the user modifies the values for P (A) 
and/or P(R), the classification rule of EQ. (7) is applied 
with the new values. 

Another variation includes not using the reject model 
606 at all, and basing all decisions on the accept model 605 
only. With this variation, the reject model 606 
is not constructed, and the formal command will be accepted 
if 

P(A)P(v|A) > Pth ... (8) 
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where Pth is a predetermined threshold. Determining the 
value for Pth is subjective and may be defined using 
trial -and-error experiments or other determination 
techniques. The designer of the system may choose a default 
5 value for Pth by trying out different values for Pth/ and 

choose a value that gives a reasonable or desirable system 
behavior. Again, the user may be permitted to 
modify the value for Pth via the user interface 100. 

Another variation includes the use of clustered accept 

10 and reject models. Referring to FIGS. GA and SB, 

schematically illustrated is an example of a. process for 
building clustered accept and reject models, in accordance 
with the present invention. In this embodiment, a plurality 
of accept models 605 and/or reject models 606 are employed. 

15 This embodiment may be adjusted to accommodate different 

sets of conditions that may lead to acceptance or rejection 
of a command. For example, a command may be rejected due to 
processing errors, e.g., because of an error made by the 
tagger 2 00 (FIG. 1) or because of an error made by the 

20 translator 300 (FIG. 1) . Further, different errors may be 

distinguished based on the type of error. For example, the 
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command itself could be wrong, or perhaps the command is 
correct but the arguments are incorrect. 

The training data from accept corpus 602 is partitioned 
into one or more accept clusters 608 by a cluster assigner 
5 607a which is coupled to the feature extractor 401a. A 

similar operation is performed for the reject corpus 603 by 
a cluster assigner 607b which is coupled to the feature 
extractor 401b. The number of clusters to be used in each 
case is a system parameter that needs to be selected by the 

10 designer of the system, which may be based on 

trial-and-error experiments or other methods. 

In one embodiment of the invention, 3 clusters for both 
the accept clusters 608 and the reject clusters 609 (the 
number of clusters does not have to be the same for the 

15 accept and reject models) . Assigning the data to one of the 

clusters can be done using rules (e.g. what kind of error 
resulted in the rejection) , or using clustering techniques 
such as a K-means clustering algorithm, which is known in 
the art. Once the data is partitioned into the desired 

20 number of clusters, model constructors 604a and 604b 

construct the desired number of accept models 605 and reject 
models 606. Each model will have its own mean vector, 
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covariance matrix and number of sentences. For example, if 
we have 3 accept models, then accept model 1 will have 
accept mean vector mAi, covariance matrix Sm and number of 
sentences nM, and similarly for accept models 2 and 3 (and 
5 reject models as well) . 

Referring to FIG. 7, a robustness checker is 
illustratively shown employing clustered models in 
accordance with the present invention. Robustness evaluator 
402 of robustness checker 400 calculates the values P(A-,) 
10 for each accept model j using: 

P(Aj) = nAj / ntctai ... (9) 

where is the number of sentences in accept cluster j , 
15 and Utotai is the total number of sentences in all of the 

accept clusters and reject clusters. Similarly, robustness 
evaluator 402 calculates the values P(Rj) for each reject 
model k using: 

P(Rk) = nRk / ntotai - - - (10) 

20 
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where Hr^ is the number of sentences in reject cluster k, 
and ntotai is the total number of sentences in all of the 
accept clusters and reject clusters. 

Given a feature vector v produced by the feature extractor 
401, the robustness evaluator 402 calculates the conditional 
probabilities P{v|Aj) and P(v|Rk) for each accept cluster j 
and reject cluster k, using: 



P(V A:) = 



4(v-'«A,)^5:A;(v-mAj) 



(11) 



^(v Rk) = 



-|(v-/nR,,r2f4(v-/n[^) 



... (12) 

10 The classification rule is as follows. A formal command with 

feature vector v is accepted if the 
raax(P (Aj) P (v|Aj) ) (over all j) > 

max(P(Rk)P(v|Rk) ) (over all k) ... (13) 



15 and rejected otherwise. If the command is to be executed, it 

is submitted to the command executor 500 (FIG. 1) as 
described above . 
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The present invention presents a robustness checker 
that can be employed by a natural language understanding 
system to determine if the user's natural language input has 
be understood correctly by the system. The robustness 
5 checker is statistically trained using the same training 

data that was used to build the natural language 
understanding system. The robustness checker calculates a 
feature vector for each input from the user, compares the 
feature vector against a set of statistical models and 
10 determines if the input should be accepted or rejected. 

When the input is accepted, the command associated with the 
input is executed. When the input is rejected, the command 
is not executed, and the user may have to resubmit the 
input . 

15 Having described preferred embodiments of a system and 

method for ensuring robustness in natural language 
understanding (which are intended to be illustrative and not 
limiting) , it is noted that modifications and variations can 
be made by persons skilled in the art in light of the above 

20 teachings. It is therefore to be understood that changes 

may be made in the particular embodiments of the invention 
disclosed which are within the scope and spirit of the 
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invention as outlined by the appended claims. Having thus 
described the invention with the details and particularity- 
required by the patent laws, what is claimed and desired 
protected by Letters Patent is set forth in the appended 
5 claims. 



10 
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WHAT IS gLAIMED IS ; 

A method for ensuring robustness of a natural 
language understanding (NLU) system comprising the steps of: 
tagging recognized words of a command input to the NLU 
5 system to associate the command with a contexts- 

translating the command to at least one formal command 
based on the tagged words; 

determining a top ranked formal command based on 
scoring of the tagged recognized words and scoring 
10 translations of the at least one formal commands- 

determining whether the top ranked formal command is 
accepted by comparing a feature vector of the top ranked 
formal command to representations of feature vectors stored 
in an accept model; and 
15 executing the top ranked formal command if accepted and 

preventing incorrect commands from execution to provide a 
robust NLU system. 



2. The method as recited in claim 1, wherein the step 
20 of determining a top ranked formal command includes the step 

of ranking formal commands based on a product of scores of 
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the tagged words and scores of translations of the at least 
one formal command. 

3. The method as recited in claim 1, wherein the step 
5 of determining a top ranked formal command includes the step 

of ranking N formal commands where N is a selectable system 
parameter . 

4, The method as recited in claim 1, further 

10 comprising the step of determining whether the top ranked 

formal command is rejected by comparing the feature vector 
of the top ranked formal command to representations of 
feature vectors stored in a reject model, 

15 5. The method as recited in claim 4, further 

comprising the step of providing the reject model by 
including representations of feature vectors of formal 
commands corresponding to words or sentences to be rejected. 

20 6. The method as recited in claim 4, wherein the 

reject model includes a cluster of models. 
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7. The method as recited in claim 6, further 
comprising the step of clustering the cluster of models 
based on at least one of mistakes in commands, mistakes in 
arguments of the command, and processing mistakes. 

5 

8. The method as recited in claim 1, further 
comprising the step of providing the accept model by 
including representations of feature vectors of formal 
commands corresponding to words or sentences to be accepted. 

10 

9. The method as recited in claim 1, wherein the step 
of determining whether the top ranked formal command is 
accepted includes the step of computing a probability of 
acceptance for the command. 

15 

10. The method as recited in claim 9, wherein the step 
of computing a probability of acceptance for the command 
includes the steps of : 

computing a probability of rejection for the command 
20 and comparing the probability of acceptance to the 

probability of rejection to determine if the command is to 
be executed. 
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11. The method as recited in claim 9, wherein the step 
of computing a probability of acceptance for the command 
includes the step of comparing the probability of acceptance 
to a threshold probability to determine if the command is to 
be executed. 

12. The method as recited in claim 11, further 
comprising the step of modifying the threshold by the user. 

13. The method as recited in claim 1, wherein the 
accept model includes a cluster of models. 

14. The method as recited in claim 13, further 
comprising the step of clustering the cluster of models 
based on at least one of mistakes in commands, mistakes in 
arguments of the command, and processing mistakes. 

15. The method as recited in claim 1, wherein the step 
of preventing incorrect commands, includes executing a do 
nothing command responsive to the incorrect commands. 
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\y. A method for building an evaluation corpus for 
checking commands in a natural language understanding (NLU) 
system comprising the steps of: 

providing a training corpus of words and sentences, the 
5 words and sentence having a user input form and a 

corresponding formal command associated with the user input 
form, at least some of corresponding formal commands 
including a do nothing command for one of incomplete and 
incorrect commands ; 
10 passing the words and sentences of the training corpus 

to the natural language understanding system to determine a 
top ranked command; 

comparing the top ranked command to the corresponding 
formal command to determine if a match exists ; 
15 if a match exists, the word or sentence is placed in 

the accept corpus, otherwise in the reject corpus; 

extracting features from the words or sentences of the 
accept corpus and the reject corpus to construct a feature 
vector for each word or sentence; and 
20 constructing an accept model and a reject model, 

respectively, from the extracted feature vectors. 
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17. The method as recited in claim 16, wherein the 
feature vectors include at least one of tagging scores for 
recognized words of the word or sentence represented by the 
feature vector and translation scores for formal commands 
5 associated with the word or sentence represented by the 

feature vector. 



18. The method as recited in claim 16, wherein the 
feature vectors include a do nothing score associated with 
10 words and sentences, the do nothing score for indicating a 

probability that the do nothing command is present for 
associated words and sentences. 



19. The method as recited in claim 16, wherein the 

15 feature vectors include a top command similarity measure for 

counting identical formal commands. 

20. The method as recited in claim 16, wherein the 
feature vectors include a parameter mismatch feature for 

20 measuring a number of command arguments in a translation of 

a command. 
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21. The method as recited in claim 16, further 
comprising the step of clustering feature vectors according 
to selected characteristics and conditions to provide at 
least one of a cluster of accept models and a cluster of 

5 reject models. 

22. The method as recited in claim 16, wherein the 
accept model and the reject model consist of mean vectors 
and covariance matrices of feature vectors representing the 

10 words and sentences and a number of words and sentences 

stored in the model . 

23/ A program storage device readable by machine, 
tangibly embodying a program of instructions executable by 
15 the machine to perform method steps for ensuring robustness 

of a natural language understanding (NLU) system, the method 
steps comprising: 

tagging recognized words of a command input to the NLU 
system to associate the command with a context; 
20 translating the command to at least one formal command 

based on the tagged words; 
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determining a top ranked formal command based on 
scoring of the tagged recognized words and scoring 
translations of the at least one formal command; 

determining whether the top ranked formal command is 
5 accepted by comparing a feature vector of the top ranked 

formal command to representations of feature vectors stored 
in an accept model; and 

executing the top ranked formal command if accepted and 
preventing incorrect commands from execution to provide a 
10 robust NLU system. 

24, The program storage device as recited in claim 23, 
wherein the step of determining a top ranked formal command 
includes the step of ranking formal commands based on a 

15 product of scores of the tagged words and scores of 

translations of the at least one formal command. 

25. The program storage device as recited in claim 23, 
wherein the step of determining a top ranked formal command 

20 includes the step of ranking N formal commands where N is a 

selectable system parameter. 
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26. The program storage device as recited in claim 23, 
further comprising the step of determining whether the top 
ranked formal command is rejected by comparing the feature 
vector of the top ranked formal command to representations 

5 of feature vectors stored in a reject model. 

27. The program storage device as recited in claim 26, 
further comprising the step of providing the reject model by 
including representations of feature vectors of formal 

10 commands corresponding to words or sentences to be rejected. 

28. The program storage device as recited in claim 26, 
wherein the reject model includes a cluster of models. 

15 29. The program storage device as recited in claim 28, 

further comprising the step of clustering the cluster of 
models based on at least one of mistakes in commands, 
mistakes in arguments of the command, and processing 
mistakes . 

20 

30, The program storage device as recited in claim 23, 
further comprising the step of providing the accept model by 
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including representations of feature vectors of formal 
commands corresponding to words or sentences to be accepted. 



31. The program storage device as recited in claim 23, 

5 wherein the step of determining whether the top ranked 

formal command is accepted includes the step of computing a 
probability of acceptance for the command. 



32. The program storage device as recited in claim 31, 
10 wherein the step of computing a probability of acceptance 

for the command includes the steps of : 

computing a probability of rejection for the command 
and comparing the probability of acceptance to the 
probability of rejection to determine if the command is to 
15 be executed. 



33. The program storage device as recited in claim 31, 
wherein the step of computing a probability of acceptance 
for the command includes the step of comparing the 
20 probability of acceptance to a threshold probability to 

determine if the command is to be executed. 
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34. The program storage device as recited in claim 33, 
further comprising the step of modifying the threshold by 
the user. 

35. The program storage device as recited in claim 23, 
wherein the accept model includes a cluster of models, 

36. The program storage device as recited in claim 35, 
further comprising the step of clustering the cluster of 
models based on at least one of mistakes in commands, 
mistakes in arguments of the command, and processing 
mistakes . 

37. The program storage device as recited in claim 23, 
wherein the step of preventing incorrect commands, includes 
executing a do nothing command responsive to the incorrect 
commands . 

A natural language understanding (NLU) system 
comprising : 

means for tagging recognized words of a command input 
to the NLU system to associate the command with a context; 
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means for translating the command to at least one 
formal command based on the tagged words; 

a robustness checker for determining a top ranked 
formal command based on scoring of the tagged recognized 
5 words and scoring translations the at least one formal 

command, the robustness checker for determining whether the 
top ranked formal command is accepted by comparing a feature 
vector of the top ranked formal command to representations 
'Z feature vectors stored in an accept model ; and 

^-=10 a command executor for executing the top ranked formal 

command if accepted and preventing incorrect commands from 
execution to provide a robust NLU system. 

39. The system as recited in claim 38, wherein the top 
r.^lS ranked formal command is ranked based on a product of scores 

of the tagged words and scores of translations of the at 
least one formal command. 

40. The system as recited in claim 38, wherein the top 
20 ranked formal command is selected from a ranked list of N 

formal commands where N is a selectable system parameter. 
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41. The system as recited in claim 38, further 
comprising an accept model for storing representations 
feature vectors of accept commands, the representations of 
feature vectors being employed to compare with a user input 

5 command to determine an acceptance probability of formal 

commands . 

42. The method as recited in claim 41, wherein the 
accept model includes a cluster of models. 

10 

43. The system as recited in claim 38, further 
comprising a reject model for storing representations of 
feature vectors of reject commands, the feature vectors 
being employed to compare with a user input command to 

15 determine a rejection probability of formal commands. 

44. The system as recited in claim 43, wherein the 
reject model includes a cluster of models. 

20 45. The system as recited in claim 38, wherein the 

robustness checker includes a feature extractor for 
extracting feature vectors from the command. 
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46. The system as recited in claim 38, wherein the 
feature vectors include tagging scores for recognized words 
of a word or sentence represented by the feature vector and 
translation scores for formal commands associated with the 

5 word or sentence represented by the feature vector. 

47. The system as recited in claim 38, wherein the 
feature vectors include a do nothing score associated with 
words and sentences, the do nothing score for indicating a 

10 probability that a do nothing command is present for 

associated words and sentences, the do nothing command being 
associated with incomplete or incorrect commands which may 
be input as a command. 

15 48. The system as recited in claim 38, wherein the 

feature vectors include at least one of a top command 
similarity measure for counting identical formal commands 
and a parameter mismatch feature for measuring a number of 
command arguments in a translation of a command. 

20 

49. The system as recited in claim 38, wherein the 
robustness checker includes a robustness evaluator for 
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determining whether the top ranked formal command is 
accepted by comparing the top ranked formal command to 
feature vectors stored in an accept model, 

5 50 ./ A natural language understanding system 

comprising: 

a corpus of rejectable commands corresponding to 
incorrect commands capable of being input by a user; and 

do nothing commands corresponding to the incorrect 
10 commands input to the system which at least one of maintain 

the system in an idle state and prompt the user to input a 
response . 
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METHOD AND SYSTEM FOR ENSURING ROBUSTNESS IN NATURAL 



LANGUAGE UNDERSTANDING 



ABSTRACT OF THE DISCLOSURE 

5 A method and system, which may be implemented by 

employing a program storage device readable by machine, and 
tangibly embodying a program of instructions executable by 
the machine to perform method steps for ensuring robustness 
of a natural language understanding (NLU) system, includes 

10 tagging recognized words of a command input to the NLU 

system to associate the command with a context, and 
translating the command to at least one formal command based 
on the tagged words. A top ranked formal command is 
determined based on scoring of the tagged recognized words 

15 and scoring translations of the at least one formal command. 

Whether the top ranked formal command is accepted is 
determined by comparing a feature vector of the top ranked 
formal command to representations of feature vectors stored 
in an accept model. The top ranked formal command is 

20 executed if accepted and incorrect commands are prevented 

from execution to provide a robust NLU system. 



Y0999-331 (8728-310) 



-47- 




loo 



Zoo 



Boo 



400 



Sentence #408 Correct 0.846709646214 

DEC: close^f Older ( name= "message log" ) 
COR: close_f older ( narne^ "message log" ) 



TDEC: [CLOSE close CLOSE] the -[FNAME message log FNAME] 
TCOR: [CLOSE close CLOSE] tile [FNAME message log FNAME] 
Complete match 



!• CLOSE the FNAME FOLDER (Tagging score: 0.873911) 



[FOLDER folder FOLDER] 
[FOLDER folder FOLDER] 




0.96887374 

0-011986108 

0-005749428 

0.0053696297 

0.0033040044 



close_f Older ( name=FNAME ) 
close_f older ( folder==current ) 
close_object { cbject=current ) 
open_f older ( naine=FNAMB ) 
close_window { window=current )^.-'-%^pi,(5C/ 
CLOSE the MSG FNAME FOLDER (Tagging score: 0. 03781 r 
0.791468 close_f older ( name=FNAME ) 

0.15737012 close_jtiessage__window( message-current ) 

0.025178693 close_f older { f older=current ) 
0.006940578 close_window{ window=current ) 
0.004540644 close_application( application^ail ) 



v3. CLOSE the FNAME (Tagging score: 0.00714398) 



1 
2 
3 
4 
U5 



0.84850997 

0.1393669 

0.0047025573 

0.002893546 

0.001893007 



close^f older { name-FNAME ) 
close_object ( object=current ) 
ORen_folder( name=FNAME ) 
close_window{ window==current ) 
close_application( application^ail ) 
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I hereby claim foreign priority benefits under Title 35, U.S. Code § 1 19(a)-(d) or §365(b) of any foreign application(s) for patent 
or inventor's certificate, or §365(a) of any PCT international application which designated at least one country other than the 
United States, or §1 19(e) of any United States provisional application(s), listed below and have also identified below any foreign 
applications for patent or inventor's certificate having a filing date before that of the application on which priority is claimed: 
O Priority Claimed ; 

_ Yes [ ] No [ ] 

_ Yes [ ] No [ ] 



(ApBcation Number) (Country) (Day/Month/Year filed) 

(Apppation Number) (Country) (Day /Month/Year filed) 

% I hereby claim the benefit under Title 35, U.S. Code, §120, of any United States application(s), or §365(c) of any PCT 
International application designating the United States, listed below and, insofar as the subject matter of each of the claims of 
thiSigpplication is not disclosed in the prior United States or PCT International application(s) in the manner provided by the first 
par^raph of Title 35, U.S. Code, §1 12, 1 acknowledge the duty to disclose information material to patentability as defmed in 
Titlp37, The Code of Federal Regulations, §1. 56(a) which became available between the filing date of the prior application 
andllhe national or PCT international filing date of this application: 
— W — - . 

(App^ation Serial Number) (Filing Date) (STATUS: patented pending, abandoned) 



(Application Serial Number) (Filing Date) (STATUS: patented, pending, abandoned) 

I hereby appoint the following attorneys: MANNY W. SCHECTER, Reg. No. 31,722; TERRY J. ILARDI Reg 29 936- 
CHRISTOPHER A. HUGHES, Reg. No. 26,914; EDWARD A. PENNINGTON, Reg. No. 32,588; JOHNE. HOEL, Reg. No 26 279- JOSEPH c' 
REDMOND, Jr., Reg. No. 18,753; KEVIN M. JORDAN, Reg. No. 40,277; STEPHEN C. KAUFMAN, Reg. No. 29,551 ; JAY P. SBROLLINI Reg 
No. 36,266; DAVID M. SHOFI, Reg. No. 39,835; ROBERT M. TREPP, Reg. No. 25,933; LOUIS P. HERZBERG, Reg. No. 41,500; DANIEL P. 
MORRIS, Reg. No. 32,053; DOUGLAS W. CAMERON, Reg. No. 31,596; LOUIS J. PERCELLO, Reg. No. 33,206; and PAUL J, OTTERSTEDT 
Reg. No. 37,411; each of them of INTERNATIONAL BUSINESS MACHINES CORPORATION, Thomas J. Watson Research Center, P.O. Box 218^ 
Yorktown Heights, New York 10598; to prosecute this application and to transact all business in the U.S. Patent and Trademark Office connected therewith 
and with any divisional, continuation, continuation-in-part, reissue or re-examination application, with Ml power of appointment and with full power to 
substitute an associate attorney or agent, and to receive all patents which may issue thereon, and request that all correspondence be addressed to: 



Frank Chau, Esq. 
F. CHAU & ASSOCIATES, LLP 
1900 Hempstead Turnpike, Suite 501 
East Meadow, New York 1 1554 
Tel.: 516-357-0091 
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I HEREBY DECLARE that all statements made herein of my own knowledge are true and that all statements made on 
information and belief are believed to be true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under §1001 of Title 18 U.S. Code and that 
such willful false statements may jeopardize the validity of the application or any patent issued thereon. 



FULL NAME OF FIRST OR SOLE INVENTOR: Ganesh N. Ramaswamv Citizenship Malaysia 

Inventor's signature: (ictAgvlv 0^ RgXAfr^gU^v^ Auysfc ,11?^ 

Residence & Post Office Address: 23 Lee Avenue, Ossining, NY 10562 



FULL NAME OF SECOND JOINT INVENTOR: Kyle A. Jamieson Citizenship 

Inventor's signature: Date: 

Residence & Post Office Address: 



FULL NAME OF THIRD JOINT INVENTOR: Jan Kleindienst Citizenship 

Inventor's signature: Date: 

Residence & Post Office Address: 
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DECLARATION Attorney Docket No. Y0999-331 (8728-310) 



AS A BELOW NAMED INVENTOR, I hereby declare that: 

My residence, post office address and citizenship are as stated next to my name. 

I beheve that I am the original, first and sole (if only one name is listed below), or an original, first and joint inventor 
(if plural names are listed below), of the subject matter which is claimed and for which a patent is sought on the invention 
entitled: 

TITLE: METHOD AND SYSTEM FOR ENSURING ROBUSTNESS 

IN NATURAL LANGUAGE UNDERSTANDING 

the specification of which either is attached hereto or indicates an attorney docket no. Y0999-331 (8728-310) , or: 

□ was filed in the U.S. Patent & Trademark Office on and assigned Serial No. , 

D and (if applicable) was amended on , 

I hereby state that I have reviewed and understand the contents of the above-identified specification, including the 
claims, as amended by any amendment referred to above. I acknowledge the duty to disclose information which is material to 
patentability and to the examination of this application in accordance with Title 37 of the Code of Federal Regulations §1.56. 
I hereby claim foreign priority benefits under Title 35, U.S. Code §1 19(a)-(d) or §365(b) of any foreign application(s) for patent 
or inventor's certificate, or §365(a) of any PCT international application which designated at least one country other than the 
United States, or § 1 19(e) of any United States provisional application(s), listed below and have also identified below any foreign 
applications for patent or inventor's certificate having a filing date before that of the application on which priority is claimed: 
: j Priority Claimed : 

Yes [ ] No [ ] 

(Ap^Ucation Number) ( Country) (Day /Month/Year filed) 

Yes [ ] No [ ] 

(Appffcation Number) (Country) (Day /Month/Year filed) 

m I hereby claim the benefit under Title 35, U.S. Code, §120, of any United States application(s), or §365(c) of any PCT 
Intejnational application designating the United States, listed below and, insofar as the subject matter of each of the claims of 
thi§:|pplication is not disclosed in the prior United States or PCT International application(s) in the manner provided by the first 
par^raph of Title 35, U.S. Code, §112, 1 acknowledge the duty to disclose information material to patentability as defined in 
Tit® 37, The Code of Federal Regulations, § 1.56(a) which became available between the filing date of the prior application 
anct iie national or PCT international filing date of this application: 

(App&cation Serial Number) (Filing Date) (STATUS: patented, pending, abandoned) 



(Application Serial Number) (Filing Date) (STATUS: patented, pending, abandoned) 

I hereby appoint the following attorneys: MANNY W. SCHECTER, Reg. No. 31,722; TERRY J. ILARDI, Reg. 29,936; 
CHRISTOPHER A. HUGHES, Reg. No. 26,914; EDWARD A. PENNINGTON, Reg. No. 32,588; JOHN E. HOEL, Reg. No. 26,279; JOSEPH C. 
REDMOND, Jr., Reg. No. 18,753; KEVIN M. JORDAN, Reg. No. 40,277; STEPHEN C. KAUFMAN, Reg. No. 29,551; JAYP. SBROLLINI, Reg. 
No. 36,266; DAVID M. SHOFI, Reg. No. 39,835; ROBERT M. TREPP, Reg. No. 25,933; LOUIS P. HERZBERG, Reg. No. 41,500; DANIEL P. 
MORRIS, Reg. No. 32,053; DOUGLAS W. CAMERON, Reg. No, 31,596; LOUIS J. PERCELLO, Reg. No. 33,206; and PAUL J. OTTERSTEDT, 
Reg. No. 37,41 1; each of them of INTERNATIONAL BUSINESS MACHINES CORPORATION, Thomas J. Watson Research Center, P.O. Box 218, 
Yorktown Heights, New York 10598; to prosecute this application and to transact all business in the U.S. Patent and Trademark Office connected therewith 
and with any divisional, continuation, continuation-in-part, reissue or re-examination application, with full power of appointment and with full power to 
substitute an associate attorney or agent, and to receive all patents which may issue thereon, and request that all correspondence be addressed to: 



Frank Chau, Esq. 
F. CHAU & ASSOCIATES, LLP 
1900 Hempstead Turnpike, Suite 501 
East Meadow, New York 11554 
Tel.: 516-357-0091 
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I HEREBY DECLARE that all statements made herein of my own knowledge are true and that all statements made on 
information and belief are believed to be true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under §1001 of Tide 18 U.S. Code and that 
such willful false statements may jeopardize the validity of the application or any patent issued thereon, 

FULL NAME OF FIRST OR SOLE INVENTOR: Ganesh N. Ramaswamv Citizenship 

Inventor's signature: Date: 

Residence & Post Office Address: 



FULL NAME OF SECOND K)INT INVENTOR: Kyle A. Jamieson 



Inventor's signature: 

Residence & Post Office Address 




5l7 Model Avenue, Hopewell, New Jersey 08525 



Citizenship USA 
Date: ^^8^/?? 



FULL NAME OF THIRD JOINT INVENTOR: 



Inye^itor's signature: 



Jan Kleindienst 



Citizenship 
Date: 



Residence & Post Office Address: 



Commonwealth of Massachusetts 
Middlesex, S.S. Date ^ { Z^i^^ 
Then personally sppear.^d the above named 

and acknowledged the foregoing instrument ^ ^ 
to be his/her ^ree act and deed, before me ^ 
RICHARD L BREWER, Notary Public 
My Commission Expires August 4, 2000 
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DECLARATION Attorney Docket No. Y0999-331 (8728-310) 



AS A BELOW NAMED INVENTOR, I hereby declare that: 

My residence, post office address and citizenship are as stated next to my name. 

I believe that I am the original, first and sole (if only one name is listed below), or an original, first and joint inventor 
(if plural names are listed below), of the subject matter which is claimed and for which a patent is sought on the invention 
entitled: 

TITLE: METHOD AND SYSTEM FOR ENSURING ROBUSTNESS 

IN NATURAL LANGUAGE UNDERSTANDING 

the specification of which either is attached hereto or indicates an attorney docket no. Y0999-331 (8728-310) . or: 

□ was filed in the U.S. Patent & Trademark Office on and assigned Serial No. , 

□ and (if applicable) was amended on 

~ ' — — ■ •* 

I hereby state that I have reviewed and understand the contents of the above-identified specification, including the 
claims, as amended by any amendment referred to above. I acknowledge the duty to disclose information which is material to 
patentability and to the examination of this application in accordance with Title 37 of the Code of Federal Regulations §1.56. 
I hereby claim foreign priority benefits under Title 35, U.S. Code §119(a)-(d) or §365(b) of any foreign application(s) for patent 
or inventor's certificate, or §365(a) of any PCT international application which designated at least one country other than the 
United States , or § 11 9(e) of any United States provisional application(s) , listed below and have also identified below any foreign 
applications for patent or inventor's certificate having a filing date before that of the application on which priority is claimed: 
n Priority Claimed ; 

^1 . Yes [ ] No [ ] 

{App^cation Number) (Country) (Day Month/Year filed) 

— 5 . Yes [ ] No [ ] 

(ApfTjcatton Number) (Country) (Day /Month/Year filed) 

Jj I hereby claim the benefit under Title 35, U.S. Code, §120, of any United States application(s), or §365(c) of any PCT 
International application designating the United States, hsted below and, insofar as the subject matter of each of the claims of 
thi^application is not disclosed in the prior United States or PCT International application(s) in the manner provided by the first 
par||raph of Title 35, U.S. Code, §112, 1 acknowledge the duty to disclose information material to patentability as defined in 
Titg37, The Code of Federal Regulations, § 1.56(a) which became available between the filing date of the prior application 
and^ lie national or PCT international filing date of this application: 

(Ap0cation Serial Number) (Filing Date) (STATUS: patented, pending, abandoned) 



(Application Serial Number) (Filing Date) (STATUS: patented, pending, abandoned) 

I hereby appoint the following attorneys: MANNY W. SCHECTER, Reg. No. 31,722; TERRY J. ILARDI Reg 29 936- 
CHRISTOPHER A. HUGHES, Reg. No. 26,914; EDWARD A. PENNINGTON, Reg. No. 32,588; JOHN E. HOEL, Reg. No 26 279 JOSEPH c' 
REDMOND, Jr., Reg. No. 18,753; KEVIN M. JORDAN, Reg. No. 40,277; STEPHEN C. KAUFMAN, Reg. No. 29,551; JAYP. SBROLLINI Reg 
No. 36,266; DAVID M. SHOFI, Reg. No. 39,835; ROBERT M. TREPP, Reg. No. 25,933; LOUIS P. HERZBERG, Reg. No. 41 500 DANIEL P 
MORRIS, Reg. No. 32,053; DOUGLAS W. CAMERON, Reg. No. 31,596; LOUIS J. PERCELLO, Reg. No. 33,206; and PAUL J. OTTERSTEDT, 
Reg. No. 37,41 1 ; each of them of INTERNATIONAL BUSINESS MACHINES CORPORATION, Thomas J. Watson Research Center, P.O. Box 218! 
Yorktown Heights, New York 10598; to prosecute this application and to transact all business in the U.S. Patent and Trademark Office connected therewith 
and with any divisional, continuation, continuation-in-part, reissue or re-examination application, with full power of appointment and with Ml power to 
substitute an associate attorney or agent, and to receive all patents which may issue thereon, and request that all correspondence be addressed to: 



Frank Chau, Esq. 
F. CHAU & ASSOCIATES, LLP 
1900 Hempstead Turnpike, Suite 501 
East Meadow, New York 1 1554 
Tel.: 516-357-0091 
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I HEREBY DECLARE that all statements made herein of my own knowledge are true and that all statements made on 
information and belief are believed to be true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under §1001 of Title 18 U.S. Code and that 
such willful false statements may jeopardize the validity of the application or any patent issued thereon. 

FULL NAME OF FIRST OR SOLE INVENTOR: Ganesh N. Ramaswamv Citizenship 



Inventor's signature: Date: 

Residence & Post Office Address: 



FULL NAME OF SECOND JOINT INVENTOR: Kyle A. Jamieson Citizenship 

Inventor's signature: Date: 

Residence & Post Office Address: 



FULL NAME OF THIRD JOINT INVENTOR: Jan Kleindienst Citizenship Czech Republic 

Inventor's signature: ^> Lg^t^ Date: 3<9 - € ^ ^9 

Resllence & Post Office Address: ' Jerevanska 2590 Kladno-Krochehlavy 270 04 Czech Republic 
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Attorney Docket No. Y0999-331 (8728-310) 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
APPLICANT(S): Ramaswamy et al. 
SERIAL NO.: Unassigned 



FILED: 



FOR: 



Concurrently herewith 

METHOD AND SYSTEM FOR ENSURING ROBUSTNESS 
IN NATURAL LANGUAGE UNDERSTANDING 



ASSOCIATE POWER OF ATTORNEY 

Please recognize FRANK CHAU, Reg. No. 34,136; JAMES J. 
BITETTO, Reg. No. 40,513; FRANK V. DeROSA, Reg. No. 43,584; and 
GASPARE J. RANDAZZO, Reg. No. 41,528; each of them of F. CHAU & 
ASSOCIATES, LLP, 1900 Hempstead Turnpike, Suite 501, East Meadow, New York 
1 1554 as associate attorneys in the above-mentioned application, with full power to 
prosecute said application, to make alterations and amendments therein, and to transact 
all business in the Patent and Trademark Office connected therewith. 



Telephone calls should be made to Frank Chau by dialing (516) 357- 



0091. 



All written communicatioiis are to be sent to Frank Chau, Esq., 
F. Chau & Associates, LLP, 1900 Hempstead Turnpike, Suite 501, East Meadow, 
New York 11554. 




International Business Machines 
Corporation 

T.J. Watson Research Center 
Route 134 and Kitchawan Road 
Yorktown Heights, New York 10598 



Maimy W. Schecter 
Registration No. 31,722 
Paul J. Otterstedt 
Registration No. 37,411 
Attorney for Applicant(s) 



